"""
飞书知识库爬虫脚本 - 增强版
专门处理多级嵌套目录结构，递归展开所有层级
支持会话保存和精确的知识库目录定位
"""

import asyncio
import os
import re
import time
import random
from typing import List, Dict, Set, Optional
from urllib.parse import urljoin, urlparse
from playwright.async_api import async_playwright, Page, Browser, ElementHandle, BrowserContext
import aiofiles
from datetime import datetime
import logging
import json

# 配置日志
logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')
logger = logging.getLogger(__name__)

class FeishuWikiCrawler:
    """飞书知识库爬虫类 - 支持多级嵌套目录和会话保存"""
    
    def __init__(self, base_url: str, output_dir: str = "feishu_wiki_docs"):
        """
        初始化爬虫
        
        Args:
            base_url: 飞书知识库的基础URL
            output_dir: 输出目录名称
        """
        self.baseUrl = base_url  # 知识库基础URL
        self.outputDir = output_dir  # 输出目录
        self.sessionDir = os.path.join(output_dir, "session")  # 会话保存目录
        self.crawledUrls = set()  # 已爬取的URL集合
        self.expandedNodes = set()  # 已展开的节点集合
        self.documentLinks = []  # 文档链接列表
        self.directoryStructure = {}  # 目录结构映射
        self.userAgents = [  # 用户代理列表
            'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36',
            'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36',
            'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36'
        ]
        
        # 创建输出目录和会话目录
        os.makedirs(self.outputDir, exist_ok=True)
        os.makedirs(self.sessionDir, exist_ok=True)
        logger.info(f"输出目录创建完成: {self.outputDir}")
        logger.info(f"会话目录创建完成: {self.sessionDir}")

    def cleanFilename(self, filename: str) -> str:
        """
        清理文件名，移除非法字符
        
        Args:
            filename: 原始文件名
            
        Returns:
            清理后的文件名
        """
        # 移除或替换非法字符
        cleaned = re.sub(r'[<>:"/\\|?*]', '', filename)
        # 替换空格为下划线
        cleaned = re.sub(r'\s+', '_', cleaned)
        # 限制文件名长度
        if len(cleaned) > 80:
            cleaned = cleaned[:80]
        return cleaned.strip('_')

    async def saveSession(self, context: BrowserContext) -> None:
        """
        保存浏览器会话状态
        
        Args:
            context: 浏览器上下文
        """
        try:
            session_file = os.path.join(self.sessionDir, "session_state.json")
            
            # 保存cookies
            cookies = await context.cookies()
            
            # 保存localStorage（如果可能）
            session_data = {
                'cookies': cookies,
                'save_time': datetime.now().isoformat(),
                'base_url': self.baseUrl
            }
            
            async with aiofiles.open(session_file, 'w', encoding='utf-8') as f:
                await f.write(json.dumps(session_data, ensure_ascii=False, indent=2))
            
            logger.info(f"会话状态已保存到: {session_file}")
            
        except Exception as e:
            logger.error(f"保存会话状态失败: {str(e)}")

    async def loadSession(self, context: BrowserContext) -> bool:
        """
        加载之前保存的会话状态
        
        Args:
            context: 浏览器上下文
            
        Returns:
            是否成功加载会话
        """
        try:
            session_file = os.path.join(self.sessionDir, "session_state.json")
            
            if not os.path.exists(session_file):
                logger.info("未找到保存的会话状态")
                return False
            
            async with aiofiles.open(session_file, 'r', encoding='utf-8') as f:
                content = await f.read()
                session_data = json.loads(content)
            
            # 检查会话是否过期（7天）
            save_time = datetime.fromisoformat(session_data['save_time'])
            if (datetime.now() - save_time).days > 7:
                logger.info("会话已过期，将重新登录")
                return False
            
            # 恢复cookies
            if session_data.get('cookies'):
                await context.add_cookies(session_data['cookies'])
                logger.info(f"已恢复 {len(session_data['cookies'])} 个cookies")
                return True
            
            return False
            
        except Exception as e:
            logger.error(f"加载会话状态失败: {str(e)}")
            return False

    async def waitForLogin(self, page: Page) -> bool:
        """
        等待用户登录完成
        
        Args:
            page: Playwright页面对象
            
        Returns:
            是否登录成功
        """
        logger.info("检测到需要登录，请在浏览器中完成登录...")
        
        # 等待登录完成的标志
        login_indicators = [
            # 知识库目录出现
            '[class*="wiki"]',
            '[class*="directory"]',
            '[class*="sidebar"]',
            # 用户头像或菜单
            '[class*="avatar"]',
            '[class*="user"]'
        ]
        
        max_wait_time = 300  # 最多等待5分钟
        start_time = time.time()
        
        while (time.time() - start_time) < max_wait_time:
            try:
                # 检查是否出现知识库内容
                for indicator in login_indicators:
                    elements = await page.query_selector_all(indicator)
                    if elements:
                        logger.info("检测到登录成功标志")
                        return True
                
                # 检查URL是否变化（登录后通常会跳转）
                current_url = page.url
                if 'login' not in current_url.lower() and 'auth' not in current_url.lower():
                    await asyncio.sleep(2)  # 等待页面稳定
                    # 再次检查是否有知识库内容
                    wiki_content = await page.query_selector('[class*="wiki"], [class*="doc"]')
                    if wiki_content:
                        logger.info("登录成功，检测到知识库内容")
                        return True
                
                await asyncio.sleep(2)
                
            except Exception as e:
                logger.debug(f"登录检测过程中的错误: {str(e)}")
                await asyncio.sleep(1)
        
        logger.error("登录等待超时")
        return False

    async def findWikiDirectoryArea(self, page: Page) -> Optional[ElementHandle]:
        """
        查找知识库目录区域
        
        Args:
            page: Playwright页面对象
            
        Returns:
            知识库目录区域的元素
        """
        # 知识库目录区域的选择器
        directory_selectors = [
            # 飞书知识库特定选择器
            '[class*="wiki-directory"]',
            '[class*="wiki-sidebar"]',
            '[class*="wiki-nav"]',
            '[class*="document-tree"]',
            '[class*="catalog"]',
            
            # 更通用的目录选择器
            '.sidebar [class*="tree"]',
            '.navigation [class*="tree"]',
            '[role="navigation"] [class*="tree"]',
            
            # 左侧面板选择器
            '.sidebar',
            '.nav-sidebar',
            '.left-panel',
            
            # 包含目录文字的区域
            '*:has-text("知识库目录")',
            '*:has-text("目录")',
            '*:has-text("首页")'
        ]
        
        for selector in directory_selectors:
            try:
                element = await page.query_selector(selector)
                if element:
                    # 验证这个区域确实包含目录内容
                    text_content = await element.inner_text()
                    if any(keyword in text_content for keyword in ['首页', '目录', '工具', 'CICD', '文档']):
                        logger.info(f"找到知识库目录区域，使用选择器: {selector}")
                        return element
            except:
                continue
        
        logger.warning("未找到明确的知识库目录区域，将使用整个页面")
        return None

    async def findExpandableNodes(self, page: Page, directory_area: Optional[ElementHandle] = None) -> List[ElementHandle]:
        """
        在知识库目录区域查找所有可展开的节点
        
        Args:
            page: Playwright页面对象
            directory_area: 知识库目录区域元素
            
        Returns:
            可展开的节点列表
        """
        expandable_nodes = []
        
        # 如果有明确的目录区域，在该区域内查找
        search_context = directory_area if directory_area else page
        
        # 飞书知识库目录的展开按钮选择器
        selectors = [
            # 具体的展开图标选择器
            'svg[data-icon="triangle-right"]',
            'svg[data-icon="caret-right"]',
            'svg[data-icon="chevron-right"]',
            
            # 通用的箭头选择器
            'i[class*="triangle"]',
            'i[class*="caret"]',
            'i[class*="chevron"]',
            'i[class*="arrow"]',
            
            # 展开按钮选择器
            'button[aria-expanded="false"]',
            '[role="button"][aria-expanded="false"]',
            
            # 飞书特定的树形控件
            '.lark-tree-switcher:not(.lark-tree-switcher_leaf)',
            '.tree-switcher:not(.leaf)',
            '[class*="tree-switcher"]:not([class*="leaf"])',
            
            # 目录项的展开按钮
            '[class*="directory"] [class*="expand"]',
            '[class*="folder"] [class*="expand"]',
            '[class*="node"] [class*="expand"]',
            
            # SVG图标（通常是箭头）
            'svg:has(path)',
            
            # 包含特定文本的可点击元素
            '[class*="clickable"]:has(svg)',
            
            # 更宽泛的选择器
            '[data-testid*="expand"]',
            '[data-testid*="toggle"]'
        ]
        
        for selector in selectors:
            try:
                if directory_area:
                    elements = await directory_area.query_selector_all(selector)
                else:
                    elements = await page.query_selector_all(selector)
                    
                for element in elements:
                    try:
                        # 检查元素是否在知识库目录区域内
                        if directory_area:
                            # 如果有明确的目录区域，检查元素是否在其中
                            bbox = await element.bounding_box()
                            area_bbox = await directory_area.bounding_box()
                            if bbox and area_bbox:
                                # 检查元素是否在目录区域内
                                if (bbox['x'] >= area_bbox['x'] and 
                                    bbox['y'] >= area_bbox['y'] and
                                    bbox['x'] + bbox['width'] <= area_bbox['x'] + area_bbox['width'] and
                                    bbox['y'] + bbox['height'] <= area_bbox['y'] + area_bbox['height']):
                                    pass  # 在区域内
                                else:
                                    continue  # 不在区域内，跳过
                        
                        # 检查元素是否可见且可点击
                        if await element.is_visible() and await element.is_enabled():
                            # 检查是否已经展开
                            expanded = await element.get_attribute('aria-expanded')
                            if expanded != 'true':
                                # 检查父元素或相邻元素的文本，确保是目录项
                                try:
                                    parent = await element.query_selector('..')
                                    if parent:
                                        parent_text = await parent.inner_text()
                                        # 过滤掉不相关的展开按钮
                                        if not any(keyword in parent_text.lower() for keyword in ['设置', 'setting', '权限', 'permission', '分享', 'share']):
                                            expandable_nodes.append(element)
                                except:
                                    expandable_nodes.append(element)
                    except:
                        continue
            except:
                continue
        
        # 去重
        unique_nodes = []
        seen_positions = set()
        
        for node in expandable_nodes:
            try:
                bbox = await node.bounding_box()
                if bbox:
                    position = (int(bbox['x']), int(bbox['y']))
                    if position not in seen_positions:
                        unique_nodes.append(node)
                        seen_positions.add(position)
            except:
                continue
        
        return unique_nodes

    async def expandNodeRecursively(self, page: Page, max_depth: int = 10) -> int:
        """
        递归展开知识库目录的所有节点
        
        Args:
            page: Playwright页面对象
            max_depth: 最大递归深度
            
        Returns:
            总展开节点数
        """
        total_expanded = 0
        current_depth = 0
        
        logger.info(f"开始递归展开知识库目录节点，最大深度: {max_depth}")
        
        # 首先查找知识库目录区域
        directory_area = await self.findWikiDirectoryArea(page)
        if directory_area:
            logger.info("已定位到知识库目录区域")
        else:
            logger.warning("未能精确定位知识库目录区域，将在整个页面中查找")
        
        while current_depth < max_depth:
            logger.info(f"=== 第 {current_depth + 1} 层目录展开 ===")
            
            # 等待页面稳定
            await page.wait_for_load_state('networkidle')
            await asyncio.sleep(1)
            
            # 在知识库目录区域查找可展开节点
            expandable_nodes = await self.findExpandableNodes(page, directory_area)
            
            if not expandable_nodes:
                logger.info("知识库目录中没有找到更多可展开的节点")
                break
            
            logger.info(f"在知识库目录中找到 {len(expandable_nodes)} 个可展开节点")
            
            # 展开当前层级的所有节点
            layer_expanded = 0
            for i, node in enumerate(expandable_nodes):
                try:
                    # 获取节点的描述信息
                    try:
                        node_info = ""
                        parent = await node.query_selector('..')
                        if parent:
                            parent_text = await parent.inner_text()
                            node_info = parent_text.strip()[:50] if parent_text else f"节点{i+1}"
                        else:
                            node_info = f"节点{i+1}"
                    except:
                        node_info = f"节点{i+1}"
                    
                    logger.info(f"  展开第 {i+1}/{len(expandable_nodes)} 个目录节点: {node_info}")
                    
                    # 滚动到节点位置
                    await node.scroll_into_view_if_needed()
                    await asyncio.sleep(0.3)
                    
                    # 点击展开
                    await node.click(timeout=3000)
                    layer_expanded += 1
                    total_expanded += 1
                    
                    # 随机延迟
                    await asyncio.sleep(random.uniform(0.5, 1.2))
                    
                    # 每展开3个节点就等待一下，让页面稳定
                    if (i + 1) % 3 == 0:
                        await asyncio.sleep(2)
                        await page.wait_for_load_state('networkidle')
                    
                except Exception as e:
                    logger.debug(f"展开节点失败: {str(e)}")
                    continue
            
            logger.info(f"第 {current_depth + 1} 层展开了 {layer_expanded} 个目录节点")
            
            # 如果这一层没有展开任何节点，结束递归
            if layer_expanded == 0:
                logger.info("当前层级没有展开任何节点，结束递归")
                break
            
            current_depth += 1
            
            # 等待页面加载新的目录内容
            await asyncio.sleep(3)
        
        logger.info(f"知识库目录递归展开完成！总共展开了 {total_expanded} 个节点")
        return total_expanded

    async def extractAllDocumentLinks(self, page: Page) -> List[Dict]:
        """
        提取所有文档链接，包括嵌套目录中的链接
        
        Args:
            page: Playwright页面对象
            
        Returns:
            文档链接列表
        """
        logger.info("开始提取所有文档链接...")
        
        try:
            # 等待页面稳定
            await page.wait_for_load_state('networkidle')
            await asyncio.sleep(2)
            
            documents = []
            
            # 多种选择器策略，针对飞书知识库
            link_selectors = [
                # 飞书知识库特定选择器
                'a[href*="/wiki/"]',
                'a[href*="/docx/"]',
                'a[href*="/sheets/"]',
                'a[href*="/slides/"]',
                'a[href*="/bitable/"]',
                
                # 目录树中的链接
                '.lark-tree a',
                '.tree-node a',
                '[class*="tree"] a[href]',
                '[class*="directory"] a[href]',
                
                # 通用文档链接
                'a[href*="/doc"]',
                'a[href*="document"]',
                
                # 标题链接
                '[class*="title"] a[href]',
                '[class*="name"] a[href]',
                
                # 所有包含href的a标签
                'a[href]'
            ]
            
            for selector in link_selectors:
                try:
                    links = await page.query_selector_all(selector)
                    logger.info(f"使用选择器 '{selector}' 找到 {len(links)} 个链接")
                    
                    for link in links:
                        try:
                            href = await link.get_attribute('href')
                            if not href:
                                continue
                            
                            # 获取链接文本
                            title = await link.inner_text()
                            if not title.strip():
                                # 尝试获取title属性
                                title = await link.get_attribute('title') or ""
                                if not title.strip():
                                    # 尝试获取aria-label
                                    title = await link.get_attribute('aria-label') or ""
                            
                            title = title.strip()
                            if not title:
                                continue
                            
                            # 处理相对链接
                            if href.startswith('/'):
                                full_url = urljoin(self.baseUrl, href)
                            elif href.startswith('http'):
                                full_url = href
                            else:
                                continue
                            
                            # 过滤有效的文档链接
                            valid_patterns = ['/wiki/', '/docx/', '/sheets/', '/slides/', '/bitable/', '/doc']
                            if any(pattern in full_url for pattern in valid_patterns):
                                # 排除一些非文档链接
                                exclude_patterns = ['edit', 'comment', 'history', 'setting', 'share']
                                if not any(pattern in full_url.lower() for pattern in exclude_patterns):
                                    # 获取文档路径（用于组织结构）
                                    try:
                                        parent_element = await link.query_selector('..')
                                        path_elements = []
                                        current = parent_element
                                        for _ in range(5):  # 最多向上查找5层
                                            if current:
                                                class_attr = await current.get_attribute('class') or ""
                                                if 'tree' in class_attr or 'directory' in class_attr:
                                                    text = await current.inner_text()
                                                    if text and text.strip() != title:
                                                        path_elements.append(text.strip())
                                                current = await current.query_selector('..')
                                            else:
                                                break
                                        path_elements.reverse()
                                        document_path = " > ".join(path_elements) if path_elements else "根目录"
                                    except:
                                        document_path = "未知路径"
                                    
                                    doc_info = {
                                        'title': title,
                                        'url': full_url,
                                        'path': document_path,
                                        'selector': selector
                                    }
                                    
                                    # 避免重复添加
                                    if not any(doc['url'] == full_url for doc in documents):
                                        documents.append(doc_info)
                                        
                        except Exception as e:
                            logger.debug(f"处理链接时出错: {str(e)}")
                            continue
                            
                except Exception as e:
                    logger.debug(f"选择器 '{selector}' 执行失败: {str(e)}")
                    continue
            
            # 去重并排序
            unique_documents = []
            seen_urls = set()
            for doc in documents:
                if doc['url'] not in seen_urls:
                    unique_documents.append(doc)
                    seen_urls.add(doc['url'])
            
            # 按路径排序，便于组织
            unique_documents.sort(key=lambda x: (x['path'], x['title']))
            
            logger.info(f"总共提取到 {len(unique_documents)} 个唯一文档链接")
            
            # 保存文档结构信息
            await self.saveDocumentStructure(unique_documents)
            
            return unique_documents
            
        except Exception as e:
            logger.error(f"提取文档链接时发生错误: {str(e)}")
            return []

    async def saveDocumentStructure(self, documents: List[Dict]) -> None:
        """
        保存文档结构信息到JSON文件
        
        Args:
            documents: 文档列表
        """
        try:
            structure_file = os.path.join(self.outputDir, "document_structure.json")
            async with aiofiles.open(structure_file, 'w', encoding='utf-8') as f:
                await f.write(json.dumps(documents, ensure_ascii=False, indent=2))
            logger.info(f"文档结构已保存到: {structure_file}")
        except Exception as e:
            logger.error(f"保存文档结构失败: {str(e)}")

    async def crawlDocumentContent(self, page: Page, doc_url: str, title: str, path: str) -> Optional[Dict]:
        """
        爬取单个文档的内容
        
        Args:
            page: Playwright页面对象
            doc_url: 文档URL
            title: 文档标题
            path: 文档路径
            
        Returns:
            文档内容字典
        """
        try:
            logger.info(f"正在爬取文档: {path} > {title}")
            
            # 导航到文档页面
            await page.goto(doc_url, wait_until='networkidle', timeout=30000)
            await asyncio.sleep(random.uniform(2, 4))
            
            # 提取文档内容
            content_selectors = [
                # 飞书文档内容选择器
                '.editor-content',
                '.doc-content',
                '.wiki-content',
                '.lark-editor',
                '[data-testid="editor"]',
                '.rich-editor',
                '.document-content',
                
                # 通用内容选择器
                '[class*="content"]',
                'main',
                '.main-content',
                '#content'
            ]
            
            content = ""
            content_html = ""
            
            for selector in content_selectors:
                try:
                    element = await page.query_selector(selector)
                    if element:
                        text_content = await element.inner_text()
                        html_content = await element.inner_html()
                        if len(text_content.strip()) > len(content.strip()):
                            content = text_content.strip()
                            content_html = html_content
                except:
                    continue
            
            # 如果没有找到内容，尝试获取整个页面的内容
            if len(content) < 30:
                content = await page.inner_text('body')
                content_html = await page.inner_html('body')
            
            # 提取发布日期和更新日期
            date_info = await self.extractDateInfo(page)
            
            # 提取图片链接
            images = await self.extractImages(page)
            
            # 提取表格信息
            tables = await self.extractTables(page)
            
            return {
                'title': title,
                'url': doc_url,
                'path': path,
                'content': content,
                'content_html': content_html,
                'publish_date': date_info.get('publish_date', ''),
                'update_date': date_info.get('update_date', ''),
                'images': images,
                'tables': tables,
                'crawl_time': datetime.now().isoformat()
            }
            
        except Exception as e:
            logger.error(f"爬取文档 '{title}' 时发生错误: {str(e)}")
            return None

    async def extractDateInfo(self, page: Page) -> Dict:
        """提取日期信息"""
        date_info = {'publish_date': '', 'update_date': ''}
        
        date_selectors = [
            '[class*="date"]',
            '[class*="time"]',
            '[class*="publish"]',
            '[class*="created"]',
            '[class*="updated"]',
            '[class*="modified"]'
        ]
        
        for selector in date_selectors:
            try:
                elements = await page.query_selector_all(selector)
                for element in elements:
                    text = await element.inner_text()
                    if re.search(r'\d{4}[-/]\d{1,2}[-/]\d{1,2}', text):
                        if not date_info['publish_date']:
                            date_info['publish_date'] = text.strip()
                        elif not date_info['update_date']:
                            date_info['update_date'] = text.strip()
            except:
                continue
        
        return date_info

    async def extractImages(self, page: Page) -> List[Dict]:
        """提取图片信息"""
        images = []
        try:
            img_elements = await page.query_selector_all('img')
            for img in img_elements:
                src = await img.get_attribute('src')
                alt = await img.get_attribute('alt') or ""
                title = await img.get_attribute('title') or ""
                if src and not src.startswith('data:'):  # 排除base64图片
                    # 处理相对路径
                    if src.startswith('/'):
                        src = urljoin(self.baseUrl, src)
                    images.append({
                        'src': src,
                        'alt': alt,
                        'title': title
                    })
        except:
            pass
        return images

    async def extractTables(self, page: Page) -> List[Dict]:
        """提取表格信息"""
        tables = []
        try:
            table_elements = await page.query_selector_all('table')
            for i, table in enumerate(table_elements):
                table_html = await table.inner_html()
                table_text = await table.inner_text()
                tables.append({
                    'index': i + 1,
                    'html': table_html,
                    'text': table_text
                })
        except:
            pass
        return tables

    async def saveAsMarkdown(self, doc_data: Dict) -> bool:
        """
        将文档保存为Markdown文件，按目录结构组织
        
        Args:
            doc_data: 文档数据字典
            
        Returns:
            保存是否成功
        """
        try:
            # 清理文件名
            filename = self.cleanFilename(doc_data['title'])
            if not filename:
                filename = f"document_{int(time.time())}"
            
            # 创建目录结构
            path_parts = doc_data['path'].split(' > ')
            clean_path_parts = [self.cleanFilename(part) for part in path_parts if part and part != "根目录"]
            
            # 创建子目录
            if clean_path_parts:
                sub_dir = os.path.join(self.outputDir, *clean_path_parts)
                os.makedirs(sub_dir, exist_ok=True)
                filepath = os.path.join(sub_dir, f"{filename}.md")
            else:
                filepath = os.path.join(self.outputDir, f"{filename}.md")
            
            # 创建Markdown内容
            markdown_content = f"""# {doc_data['title']}

**文档路径**: {doc_data['path']}
**文档链接**: {doc_data['url']}
**爬取时间**: {doc_data['crawl_time']}
"""
            
            if doc_data['publish_date']:
                markdown_content += f"**发布日期**: {doc_data['publish_date']}\n"
            
            if doc_data['update_date']:
                markdown_content += f"**更新日期**: {doc_data['update_date']}\n"
            
            markdown_content += "\n---\n\n"
            
            # 添加主要内容
            if doc_data['content']:
                markdown_content += doc_data['content']
            
            # 添加图片信息
            if doc_data['images']:
                markdown_content += "\n\n## 文档图片\n\n"
                for i, img in enumerate(doc_data['images'], 1):
                    alt_text = img['alt'] or img['title'] or f"图片{i}"
                    markdown_content += f"![{alt_text}]({img['src']})\n\n"
            
            # 添加表格信息
            if doc_data['tables']:
                markdown_content += "\n\n## 文档表格\n\n"
                for table in doc_data['tables']:
                    markdown_content += f"### 表格 {table['index']}\n\n"
                    markdown_content += f"```\n{table['text']}\n```\n\n"
            
            # 保存文件
            async with aiofiles.open(filepath, 'w', encoding='utf-8') as f:
                await f.write(markdown_content)
            
            logger.info(f"文档已保存: {filepath}")
            return True
            
        except Exception as e:
            logger.error(f"保存文档时发生错误: {str(e)}")
            return False

    async def run(self) -> None:
        """运行爬虫的主要方法，支持会话保存和恢复"""
        logger.info("开始运行飞书知识库爬虫（多级嵌套目录版本，支持会话保存）...")
        
        async with async_playwright() as p:
            # 启动浏览器
            browser = await p.chromium.launch(
                headless=False,  # 设置为True可以无头模式运行
                args=['--no-sandbox', '--disable-dev-shm-usage', '--disable-blink-features=AutomationControlled']
            )
            
            try:
                # 创建持久化浏览器上下文
                context = await browser.new_context(
                    user_agent=random.choice(self.userAgents),
                    # 设置一些更真实的浏览器选项
                    viewport={'width': 1920, 'height': 1080},
                    ignore_https_errors=True
                )
                
                # 尝试加载之前保存的会话
                session_loaded = await self.loadSession(context)
                
                # 创建新页面
                page = await context.new_page()
                
                # 导航到知识库首页
                logger.info(f"正在访问: {self.baseUrl}")
                await page.goto(self.baseUrl, wait_until='networkidle', timeout=30000)
                
                # 等待页面完全加载
                await asyncio.sleep(3)
                
                # 检查是否需要登录
                current_url = page.url
                page_content = await page.content()
                
                # 判断是否在登录页面或需要登录
                need_login = (
                    'login' in current_url.lower() or 
                    'auth' in current_url.lower() or
                    'sign' in current_url.lower() or
                    '登录' in page_content or
                    'login' in page_content.lower()
                )
                
                # 如果加载了会话但仍然需要登录，说明会话已失效
                if session_loaded and need_login:
                    logger.info("检测到会话已失效，需要重新登录")
                    session_loaded = False
                
                # 如果需要登录
                if need_login:
                    login_success = await self.waitForLogin(page)
                    if not login_success:
                        logger.error("登录失败或超时，无法继续爬取")
                        return
                    
                    # 登录成功后保存会话
                    await self.saveSession(context)
                    logger.info("登录成功，会话已保存")
                elif session_loaded:
                    logger.info("成功使用之前保存的会话状态")
                else:
                    logger.info("无需登录，直接访问知识库")
                
                # 确保页面完全加载
                await page.wait_for_load_state('networkidle')
                await asyncio.sleep(5)
                
                # 递归展开知识库目录
                expanded_count = await self.expandNodeRecursively(page, max_depth=15)
                logger.info(f"知识库目录展开完成，共展开 {expanded_count} 个节点")
                
                # 提取所有文档链接
                documents = await self.extractAllDocumentLinks(page)
                
                if not documents:
                    logger.warning("未找到任何文档链接，请检查页面结构")
                    return
                
                logger.info(f"准备爬取 {len(documents)} 个文档")
                
                # 爬取每个文档
                successful_count = 0
                failed_count = 0
                
                for i, doc in enumerate(documents, 1):
                    try:
                        logger.info(f"进度: {i}/{len(documents)} - {doc['path']} > {doc['title']}")
                        
                        # 爬取文档内容
                        doc_data = await self.crawlDocumentContent(page, doc['url'], doc['title'], doc['path'])
                        
                        if doc_data and len(doc_data['content']) >= 30:
                            # 保存为Markdown文件
                            if await self.saveAsMarkdown(doc_data):
                                successful_count += 1
                            else:
                                failed_count += 1
                        else:
                            logger.info(f"跳过空白文档: {doc['title']}")
                            failed_count += 1
                        
                        # 随机延迟，避免被封禁
                        await asyncio.sleep(random.uniform(1, 3))
                        
                    except Exception as e:
                        logger.error(f"处理文档 '{doc['title']}' 时发生错误: {str(e)}")
                        failed_count += 1
                        continue
                
                # 生成爬取报告
                await self.generateCrawlReport(successful_count, failed_count, len(documents))
                
                logger.info(f"爬取完成！成功: {successful_count}, 失败: {failed_count}, 总计: {len(documents)}")
                logger.info(f"文档保存在: {self.outputDir}")
                
                # 保存最终的会话状态
                await self.saveSession(context)
                
            except Exception as e:
                logger.error(f"爬虫运行过程中发生错误: {str(e)}")
            finally:
                try:
                    await browser.close()
                except:
                    pass

    async def generateCrawlReport(self, successful: int, failed: int, total: int) -> None:
        """生成爬取报告"""
        try:
            report = {
                'crawl_time': datetime.now().isoformat(),
                'total_documents': total,
                'successful_documents': successful,
                'failed_documents': failed,
                'success_rate': f"{(successful/total*100):.1f}%" if total > 0 else "0%",
                'total_expanded_nodes': len(self.expandedNodes),
                'base_url': self.baseUrl
            }
            
            report_file = os.path.join(self.outputDir, "crawl_report.json")
            async with aiofiles.open(report_file, 'w', encoding='utf-8') as f:
                await f.write(json.dumps(report, ensure_ascii=False, indent=2))
            
            logger.info(f"爬取报告已保存到: {report_file}")
        except Exception as e:
            logger.error(f"生成爬取报告失败: {str(e)}")

async def main():
    """主函数"""
    # 配置参数
    BASE_URL = "https://my-ichery.feishu.cn/wiki/UkYtwGxOtieErfkzUKkcjx48nib"
    OUTPUT_DIR = "feishu_wiki_docs"
    
    # 创建爬虫实例并运行
    crawler = FeishuWikiCrawler(BASE_URL, OUTPUT_DIR)
    await crawler.run()

if __name__ == "__main__":
    asyncio.run(main())